Systematics Quantification Paper

with Chris Palmer and Jacob ("Jake") David Weiling Lee

Paper describing breaking down systematics in terms of Fourier modes

Make up envelopes used to constrain the coeffients Plotting the changes

Logs

24 July 2025 Thursday

  • s2 Method does not assume the Assumption, Your setup of the toys does
  • s5: Envelop priors probably dont matter much.
  • Discussed changing the "envelope fit" to have a shape fit without inflating the uncertianties

20 June 2025 Friday

  • Understand better whats going on.
  • Had some reasonable suggetions.

Reviewing Slide 03 June 2025 Tuesday

  • Always use a different underlying truth.
  • s7: How many parameters does the F-test prefer here?
  • s8: Can you show some of the background-only fits? What are the p-values? I'm not sure these bands computed in this way are that meaningful. You're taking the spread of fits to different underlying truths. Some of the variance in the observed fits will be due to variance in the underlying truths. It may be better here to fix the underlying truth PDF and perform N-fits from samples derived from it; then the observed variance reflects the fits, not the problem specification.
  • s9: Does the F-test show that 5 is enough?
  • s11: To test the fits with the nuisance parameters, can you try fitting to background-only data? It would be good to do this N times and show the pulls on the different nuisance parameters and the fit quality. When you do the signal plus background fits, can you show the signal plus background fit in the plot? The pulls shown in the ratio still seem off to me when adding green and orange… It would also be good to show the pulls on the nuisance parameters… You can create a separate figure or somehow add it to the bottom left.
  • s12: (Checking if I understand correctly, the mean values of the nuisance parameters should all be consistent with 0, right?) It's odd that the normalization is off. The first five bins, which dominate the total yield, are all high. Are you allowing the overall normalization to float? For example, is func0 on slide 4 a constant?
  • s13: Can you overlay the shapes of the priors for the five different nuisance parameters? It would be interesting to see how the effective shapes used in s13 compare to the Gaussians.

Review Slides 31 May 2025 Saturday

  • s4: I think some small correlation is okay. Best to keep orthogonalization with respect to the expected background.
  • s5: It looks like overfitting to me. How do you determine how many modes to include? Can you also quote X²/dof or, better yet, Prob(X², nDoF)? My guess is that most of the 12(!) DoF are not statistically significant in this example.
  • s6: Can you remind me how the envelope and model uncertainties are defined?
    • Is the envelope the range of many different pseudo datasets?
    • What do you do again for the model uncertainties ?
  • s7: Where do you get the "observed data" in the signal + background fits? Is it drawn from pseudo data with different true parameters or fluctuated from the expected background? (I assume there is no signal injected here.)
  • s8: Can you also quote the p-value of the signal + background fit and the background-only fit? e.g.: what is the Prob(X², nDoF)? This fitted signal strength is unbiased, but it looks like the background modeling is poor.
  • s9: What do you mean by "each background-only" fit? What distributions are you fitting ? Is it a) Poisson-fluctuated "observed" data or b) toys with different underlying PDFs? Something else? I'm not sure why you need to use KDEs here. I would just assign Gaussian constraints as a first pass.
  • s10: I'm lost. What does the plot show? Are these 2 of the 5 fitted coefficients? Why 5? Shouldn't they have uncertainties on them?
  • s11: The pull ratio on the left looks wrong: the last 6 bins are all high, but the associated pulls are not all positive. Also, what uncertainty goes into the definition of the pull here? Again, you should quote the fit probability; it would also be good to report the pulls on the nuisance parameters. Maybe this is what you are attempting with the middle plots? It would be good to compare the size of the signal uncertainty here with s7. Did you look into the correlation of the signal shape with the other NPs?

13 May 2025 Tuesday

  • One toy as the pseudodata
  • Another toy as the background model
  • Fit the normalization using the envalope as the uncertainties

22 April 2025 Tuesday

  • Many toys with different truth varations
    • How to extend to
  • Do you decorrelate the basis function wrt nominal
  • Bands vs individual 1d distributions

Follow-ups

Links:

202504240720

Backlinks